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ABSTRACT 

The objective of this study of California projects, 
funded under Title I of the 1965 Elementary Secondary Education A.ct, 
was to shed additional light on the cost-benefit and "critical mass" 
issues which are central to today's compensatory education planning 
and policy making. It covers reading and math programs in grades one 
through twelve. The study sample included all schools in California 
which reported both achievement gains and expenditures. The data 
analysis utilized statistical controls such as the removal of the 
effects of regular per-pupil expenditures when the impact of 
supplementary expenditures was assessed. Saturated schools—thosje 
with 75 percent or more of the pupils eligible for Title I 
participation—were found to differ significantly from unsaturated 
schools with the latter showing greater gains in both reading and 
math and greater Title I per-pupil expanditures. Within the 
unsaturated schools, however, there was no relationship between 
achievement gains in either reading or math and any combination of 
regular and supplementary expenditures. In saturated schools, a 
significant relationship was found between achievement gains and 
Title I per-pupil expenditures for reading but not for math. The 
expenditure differences accounted for about ten percent of the 
variation in achievement. (Author/JM) 
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EXECUTI^ SmmRY 



The objective of the present study was to shed additional light 
on the cost'-benefit and "critical mass" Issues which are central to 
today's compensatory education plannin;^; and policy making. Because it 
depended entirely on existing data made available by the California 
State Department of EducatiDnp the study Is subject to many of the 
shortcomings of its predecessors, Its scope i however, is somewhat more 
comprehensive (reading and math programs in grades 1 through 12), its 
sampling somewhat more representative (all rchools in the state which 
reported both achiavement gains and expanditures) , and its statistical 
controls yomewhat mure adequate (Uia efiects of regular per-pupil b%- 
penditures were statistically removed when the impact ^of supplementary 
expenditures was assessed) than has been typical of studies in this 
field. 

Saturated schools (those with 75% cr more of the pupils eligible 
for Title I participation) were found to differ significantly from un^ 
saturated schools with the latter showing greater gains in both read- 
ing and math and greater Title I per-pupil expenditures, Within the 
unsaturated schools, however, there was no relationship between achieve^ 
ment gains in either reading or math and any contiination of regular and 
supplementary expenditures. In saturated schools, a. significant rela- 
tionship was found between achievement gains and TltlR I per-pupil 
expenditures for reading but not for math. The expenditure differences 
accounted for about 10% of the variation in achievement. 

Achievement gains were generally greater than mon'ch tut month at 
all grade levels and averaged 1.4 months par month for both reading 
and math in unsaturated schools* This finding, however^ was inconsistent 
with the fact that grade-equivalent status at the time of pretesting was 
corislstently below the so-called disadvantaged norm of .7 months per 
month. At the end of sixth grade, performance was at the beginning 
fifth grade level — almost exactly the normal eKpectation for disad^ 
vantaged children. This finding suggests either that children tend 



not to continue in Title 1 projects from year to yaar or that acceler- 
ated school-'year learning is matched by sccelerated losses over the 
summer. 

No evidence could be found suggesting the axis tenet of any kind of 
critical mass. For the most partj there was no relationship between 
expenditures and gains* Where such relationships did exists they 
appeared to be consistent over the entire ranges of gains and 
expandituras* 

It was poluted out that the deficiencies known to exist in the 
data wuld tend to attenuate or obscure whatever real relationships 
might exist* This would imply that the true cost-benefit relationship 
might be somewhat stronger than that which was observed * There was 
no Indication^ howeverj that better data would uncover a critical mass 

Some evidence was found that supplementary per-pupil expenditures 
are not proportional to the educational needs of pupils* For math 
projects In saturated schools , a significant negative relationship was 
found* 
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INTRODUCTION 



A leading consumer organization advertises its monchly product 
evaluation publication with the question, "Would you pay 1480.00 for n 
color T.V. set if you could get better perf orniance- for $80.00 less?" 
- implylni that there are major qualltntlve differences Lmong productB 
which are independent of price. The old saw, "You gat what you pay 
for", represents the other side of the coin. 

Compensatory education, as a consumer product, has been the subject 
of a similar. difference of opinion, and both' sides cite empirical evi» 
dance (sometimes without appropriate scientific detachment) to support 
their stand (Welsh, 1972) . Proponents of the existence of a relation- 
ship between expenditures and beriefitg in educational programs for the 
disadvantaged (DHEW, 1972) are able to quote a variety of studies which 
have found such a relationship. In all cases, as is pointed out by the 
other side however, there has been some question as to the reliability 
and/or validity of the data on which they were based. 

Wargo, Tallmadge, Michaels, Lipe, and Morris (1972) were unab .e to . 
find nationally representative data indicating that Title 1 had had any 
impact whatsoever on program participants to say nothing of finding 
a relationship between costs and benefits. Some successful state 
programs and a larger number of local projects were identified, but 
the preponderance of meaningful data at these levels failed to support 
the existence of a cost-effsctiveness relationship. Despite the lack 
of empirical evidence, however, the authors concluded that "it seems 
likely that some positive relationship does exist [p.. 193]." 

Perhaps the most frequently cite, relevant study (e.g., by President 
Nixon in his message to Congress concernini the Equal Educational 
Opportunities Act of 1972) Is the so-called "California Follow-up," a 
cooperative, two-day analysis carried out by USOE's Bureau of Elementary 
and Secondary Education and the California Diviaion of Compensatory 
Education using a sample of FY 69 Title I projects in California. This 
unpublished analysis started with cost and achievement data on 709 
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local Title I projects. From these projecta, which did not include 
any from the state's ten largest school districts, a subaample of 71 
was selected which Included all projects with a primary emphasis on 
rending and which fell into one of the following three cognitive growth 
categories^ (a) at least 2 months grade-^equivalent gain per month of 
project eKposure, (b) 1,5 to 1,9 months per month, and (c) less than 
1 month per month of exposure (a fourth growth category, 1,0 to 1,4 
months per month, was sKcluded from the analysis for reasons of economy). 

The California Follow-up study showed a clear and rather strong^ 
relationship between gains and per-pMpil Title I expenditures. It 
acknowledged, however, that the sample of projects studied was not 
representative and that the fiiidings were not generali2able to the 
state as a whole, ^^ether the sample of projects was representative 
or not, however, is less important than the two questionsi (a) *-01d 
costs and benefits covary?'* (they did), and (b) "Were increased expen-- 
dltures responsible for the covariation?'- (alternative hypotheses may 
be tenable) . 

One alternative to the causal relationship between expenditures ' 
and gains is the hypothesis that the better learners within the Title 
I population receive larger per-pupil eKpendltures than the jess able. 
This hypothesis has some plausibility since (a) the '-formula" for 
determining Title 1 allocations is tied to regular per-pupil expendi" 
tures in such a way that schools with above-- average regular expendi- 
tures will receive above-average Title I per^pupll funding and (b) 
such schools (in 1969, at least) were in above-average socioeconomic 
status areas. It has, of course, been found repeatedly that learning 
ability is related to the socioeconomic status of schools (Coleman, 
Campbell, Ho^son, McPartland, Mood, Welnfeld, & York, 1966; Chalupsky, 
Danoff, CoUj, Roberts, Rodabaugh, and Weisgarber, 19 72). 

A Chi Square analysis performed by th& author of this report on a 
Lhree^by-three contingency table representing the three cognitive 
growth categories and three levels of per-pupil expenditure (0 - $149, 
$150 - $249, and $250 - $649) was statistically significant at the 
•001 level, ^ 



Several .other cost-^effectivaness studies were coiuiucted in the State' 
of CfllifDrnia. One, undertaken by the Office of the Los M^elc-s County 
Superintendent of Schools (1970) examined the impact of variations in 
funding on Title X reading projects for second and third grado children. 
This rather well^controlled study concluded that "there was litcl« 
evidence of a relationship bett.een .cost and reading ach tavemmit [p. 
1^]." Klesllng (1971), on the other hand, found that the amount of 
instruction given by trained reading speclallsta was consistently 
related to achievenient gains. In a later paper (1972), he estimated 
that the amount of such instruction which would be required to bring 
disadvantaged children from a learning "rate" of .7 months per month 
up to the national norm would coat approximately |300 per pupil. 

Klesllng 's research exemplifies an approach rarely taken in examin- 
ing cost-benefit relatlonshipa . He began with th« identification of 
an instructional process which "worked" and then related costs to vary- 
ing amounts of treatment employing this process. It is far more common 
for research in this field to make the implicit assumption = that the 
spending of money is, in itself, an Instructional treatment. 

It seems reasonable to assume that differences In spending effec- 
tivaness will occur from project to project. If it can be assumed that 
these differences are randomly distributed throughout the population 
of projects, they would act as "noise" in the system or, in statistical 
terminology, "error variance." They would attenuate but ■ should not 
completely obscure whatever cost-effectiveness relationships exist. 

There are other attenuating factors as well, including Inadequate 
and unstandardiMd accounting practices, use of a variety of different 
achievement tests, and non-uniform testing intervals. Taken together 
these factors generate a substantial amount of random error. 

In addition to the random influences, are variables which are known, 
or suspected, to vary systematically with either expenditure, or achieve- 
ment, or both. These confounding variables need to be considered in 
study design because, unlike the random influences, they may generate ' 
specific biases — not noise. 
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Systematic Influences are not nacesaarily important where samples 
are large and representative since, under these conditions, they do not 
introduce bias. Even where this is the case, however, it is generaUy 
desirable to be able to assess the impact of LV.ch influences. This is 
particularly true where they function as rroderator variables affecting' 
relationships among other variables. The finding of Chalupsky et al. 
(19 72) that the relationahips of achievement with school, teacher, and 
pupil characteristics were signif Icantly different for low socioeconomic 
status pupils than for high' socioeconoiuic status pupils underscores 
the importance of this consideration. When socioeconomic status was not 
coiisidered in this way, most other relationships tended to disappear. 

The purpose of all this discussion is merely to point out that the 
cost benefit issue is a complex one. Many factors are at least pot.?.n- 
tially relevant and could act either so as to obscure real relationships 
or to introduce spurious relationships depending on how the sampling 
and other aspects of experimental design are handled. 

To date there has been no adequate analysis of the relationship 
between cost and effectiveness In California or anywhere else. I^or 
does it seem likely that the issue will be adequately dealt with in the 
near future, as to do so would be an undertaking of vast proportions. 

On the other hand, each, study completed to date, if interpreted 
with full consideration of its strengths and weaknesses, expands our 
knowledge relative to this important issue. Each study, furthermore, 
suggests new ways of gathering additional' information an^ addressing 
new aspects of the overall question. 

The study reported herein aspires to no more. It is perhaps most 
nearly akin to the California Follow-up although its findings are not 
entirely compatible with those of that study. The differences, however, 
appear to ba explainable in terms of differences In, sampling and 
analytical procedures. 
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METHODOLOGY 

The present study was designed to make usn of exiHting data. Its 
uhdartaking \;as motivated by the ongoing controvarrjy pv^sr the cost-- 
effectiveness issue in compensatory education in the hone that additional 
relevant Information could be cotnpiled ranidly and inQXpensival^' . 

Any study of this type is seriously limited witii respect to the 
scientific rigor which can be brought to bear on the issues. The 
exact nature of these limitations is discu&sed below and must be kept 
in mind when interpreting the study *s findings. 

The Data and their Limitations 

The State of California collects^ via questionnaire ^ substantial 
amounts of data each year regarding Title I projects from participating 
educational agencies • These data are supplied at tiie scl^ool level if 
schools are '^saturated" ™ that is, if 75% or more of tiieir pupils are 
Title I eligible. Reporting is at the school^dis trict level if schools 
are unsaturated^ 

Achievement data are collected by grade level separately for read^ 
ing and mathematics and consist of (a) name of testj (b) form, (c) 
level, (d) months between pre- and posttest, (e) number of students 
receiving both pre- and posttestSj. (f) median pretest grade equivalent ^ 
(g) median posttest grade equlvalenti (h) posttest minus pretest grade 
equivalentj (i) pretest mean scale score, ( j ) posttest mean scale scores 
and (k) posttest mean scale score minus pretest mean scale score. 
Because of the yarlety of tests used and the clear non-equivalence of 
scaled scores, the researcher interested in state'-wlde analyses has no 
choice but to rely on the median grade-equivalent data, even though the 
deficiencies of such scores have been well documented by Kiesling (1970) 
and others * ■ 

On the same questionnaire , schools (if saturated) or school districts 
(if unsaturated) provide estlmatas^^ of the (a) Title I, (b) Special 

School accounting procedures are such that, while Title I and other 
supplementary eKpenditures are usually well documented by project , thev 
are not generally broken down by project component (reading^ math, etc.)*' 



5 



Teacher Employment Programp (c) Miller--Unruh Basic Reading Act, (d) 
District Funds, and (e) other supplementary funds spent for reading and 
(separately) for math iprojects. The numbers of participants by grade 
level by target school are also called for on the questionnaire ^ but 
these data are of little use for studies of this type since costs are. 
not similarly broken down. 

The questionnaire incdrporates no provision for reporting regular 
per-pupil eKpendltures . These data are, however, available in a separate 
document, California Public Schools, Selected Statistics , an annual pub- 
lication of the State Department of Education. Unfortunately for the 
purposes of this study, while all other data were drawn from FY 72, 
regular per^pupil expenditure data had to be taken from FY 71 as the 
1972 data had not been published at the time the analyses were undertaken. 

Ideally, for an analysis of the type described herein, all data 
should be available on a per-^pupil basis. Furthermore, achievement data 
should be collected using a single (within grade level) standardized 
j.nstrumenc administered on the same dates statewide for both pre- and 
poattests* Standard scores should be employed. 

Clearly none of these desirable conditions was met. At best, some 
achievement data were available by grade level within (saturated) schools. 
The remaining achievement data were by grade level within school district • 
but different tests were used^ administered at different times and " 
Intervals^ and with scores eKpressed in grade equivalents ~ a practice 
particularly inappropriate for non-average students. 

In the cost domain, the situation was still less desirable with no 

data available by grade level and only some (supplementary expenditures 

in saturated schools) available at the school level. Moat cost dataware 
available only at the district level, 

AH of these shortcomings in the existing data should have the 
effect of Attenuating the evidence of whatsvir cost^benef it relationships 
may really exist* They should not produce any aystematic biases resulting 
in the emergence of spurious relationships. 



Perhaps the most severe of the shorccomings are those relating to 
the cost data. Even these, however, should not obscure any significant 
relationships which may exist i£ certain assumptions are valid. The 
' lack of cost data by grade level, for exainple, is of no import if tho 
analyses are conducted separately for each grade and if the pattern 
of par-pupil expenditure variation by grade level is similar from school 
to school, To establish correlational relationships between expenditure 
and gains it is not 'necessary to know the exact values of either — only 
their relative values from one school to another. 

This same argument applies to the fact that the regular per-pupil 
expenditure data used In this study were from JY 71 while all other 
data were from FY 72. As long as each district maintained nearly the 
same position relative to other districts on the per-pupll expenditure 
continuum both years ^ the consequences of this situation she uld be of 
little significance regardless of state-^wide increases or decreases. 
Procedures 

A computer tape was purchased from the California State Department 
of Education which contained the following ESEA Title I evaluation datBi 

(a) number of participants by grade level in reading and matb^^^projects , 

(b) reading test data by grade level (see above for description), (c) 
math test data by grade level, (d) supplementary per--pupil expenditure 
estimates for reading and math components broken down by funding source 
(Title I, Special Teacher Employment Program, etc.) and (e) coding infor^ 
mation identifying school and/or districts and whether saturated or un- 
saturated* The tape presumably was a copy of that used by the State 
Department in preparing Iti annual Title 1 evaluation report but with 
data not directly related to the present study deleteci. 

In preparation for analysis, special data files were created which 
Inciuded the following ten variables in addition to identifying codes t 
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Variable Descrlntlon 

1 Saturated - Unsaturated 

2 ' Pretest Median Grade . Equivalent 

3 Post test Median Grade Equivalent 

^ Average' Monthly Gain [ (Var 3 - Var 2) + 

months bet\^een testing] 

.5 Nuniber of Students with both Pre- and Posttests 

6 Regular Per-pupil Expenditure (merged from 

keypunched cards) 

=•= 

'7 Title I Per^pupil Expenditure (Title I 

eKpenditure'T number of participants)/ 

8 Title I plus Special Teacher Employment 

Program Per--pupil Expenditure (Title I + 
STEP expanditures r number of participants) 

9 To^jal Supplementary Per^pupil Expenditure 

^ (Title I + STEP + Miller-Unruh + District 
+ "other supplementary" expenditures t 
nunter of participants) 

10^ Total Per-pupil Expenditure (Var 6 + Var 9) 

Means and standard deviations of these variables and their inter- 
correlations were computed using the BW 02D program. This program was 
selected because it produces scatter diagrams of selected pairs of vari= 
ables thereby providing immediate visual evidence regarding the "critical 
mass"" issue. 

Analyses were by grade levels separately for reading and math^ and 
separately for saturated and unsaturated schools. Forty-eight sets of 
statistics (12 grade levels x 2 subject matters x 2 types of schools) 
were thus obtained, < These statistics as well as additional analyses 
performed upon them are discussed in subsequent chapters* 
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RESULTS 



Ai stated in the previous chapter, analyses were carried out in 
such a way that there were four main groups dafined by subject matter 
(reading and math) and type of school (saturated and unsaturated) 
Within these four groups, separate analyses were conducted at each 
grade level yielding. a total of 48 sets of Btatistics. Tables 1 through 
4 present the mean achievement and coat figures for each of the 48 
analyses. It. should be noted that sample sizes are very small for grades 
7 through 12 in saturated schools . The representativeness of achieve- 
ment and cost means (for both reeding and math) for these grade levels 
is clearly Inadequate, and the values presented in the tables may be 
grossly different from state-wide averages. 

Of, interest in Tables 1 through 4 is the fact that nearly all of 
the achievement gains (at least where sample sizes are adequate) exceed 
month for month. At the same time, however, test results show that the 
discrepancies between grade levels and grade equivalent scores increase 
with Increasing grade level. This apparent Inconsistency is discussed 
in more detail later. 

Also of note, is the relationship between grade level and average 
monthly gain which is quite strong for reading projects in unsaturated 
schools (r = .82) and also substantial for math projects in unsaturated 
schools (r = .59) but which is not found in saturated schools (r s ,10 
for reading and r « -.34 for math) , The cause of this phenomenon is 
not entirely clear. It should be noted, however, that the discrepancy 
between grade level and pretest grade equivalent is substantially greater 
(the pupils are farther behind grade level) In the unsaturated schools. 
Furthermore, the discrepancy is greater for unsaturated reading than 
for unsaturatad math projects. Taken togather these facts would tend 
to suggest that the observed relationships are due to operation of the 
so-called regression effect (cf. Gage, 1963). With the correlations 
baaed as they are on a small sample size (12 grade levels), however, and 
with thfr further complication that soma of the data points are themselves 
suspect, overinterpretatlon of these findings is to be avoided. 
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The data presented in Tables 1 through k are summarised in Table 
5 which presents weighted averages of the statistics computed across 
grade levels . This table shows clearly that Title I and Total Supple- 
mentary per-pupil eKpenditures are suhstantiairy higher in unsaturated 
than in saturated schools although there is no difference in Regular 
per-pupil eKpanditures* It is also apparent that gains are greater in 
unsaturated schools* 

TABLE 5 



Achievement and Expenditure Means 


Suramarized * 


icross Grade 


Levels 




Reading 


Math 


Saturated 


Unsaturated 


Saturated 


Unsaturated 


Average Monthly Gains 


1.04 


1.42 


1.29 


1.40 


Pupils per School/District 


117 


124 


515 


521 


Regular $ per Pupil 


817 


801 


817 


794 


Title r $ par Pupil 


76 


205 


59 


156 


Title I plus STEP 


83 


206 


65 


156 


Total Supplementary $ 


104 


257 


86 


164 


Total $ par Pupil 


921 


1058 


903 


958 



One's immediate first impressiofi from the figures presented in 
Table 5 li that Increased Title I per-pupil expenditures result in 
greater achievement benefits » Upon closer examination the picture is 
not nearly so clear. As shown in Figures 1 and 2, for example^ those 
Title I participants in unsaturated^ schools who show the greatest growth 
rates generally begin each school year with a somewhat lower pretest 
status than their counterparts in saturated schools and are not notably 
superior to them at the. end of the year. At the end of the sixth grade 
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(the last point at which sample sizes are adequate), all groups are 
nearly equal and all groups have averaged, since beginning first grade, 
a growth rate approximately equal to the disadvantaged norm of .7 years 
growth per year in school (Reading, Unsaturated = .70; Reading, Saturated 
= .68; Math, Unsaturated = .73; Math, Saturated * .74). 

There is some danger in interpreting Figures 1 and 2 as if they 
represented longitudinal data. They do not — the data are cross sec- 
tional and each year's growth is represented by a different sample of 
pupils. For this reason it is not strictly legitimate to talk about 
losses over the summer. We do not know how those children represented 
by each pretest point on the figures scored at posttest time the year 
before. Still, it seems reasonable to assume that many, and perhaps 
most, of the children served by Title I in the sixth grade this year 
were also served last year in the fifth grade and in earlier grades 
and years as well. Until data are acquired over at least a 12-month 
interval (ideally from posttest one year to posttest the following year), 
questions of this sort must remain unanswered. 

Intercorrelations among the ten variables considered in this study 
were also calculated, although variable 1 (Saturated vs. Unsaturated 
schools) was not meaningful within groups. (Variable 10,^ Total Per- 
pupll Expenditures, has also been eliminated from most presentations 
since it is completely redundant with the sum of variables 6, Regular 
Per-pupll Expenditures, and 9, Total Supplementary Per-pupil Expenditures). 
•Visual examination of the 48 correlation matrices revealed some grade- 
level differences but none which would contraindlcate pooling the corre- 
lations-across grade levels within groups. 

Before considerini the matrices obtained by pooling across grade 
level, some mention should be made of the differences which were obBerved 
between grades^. The most interesting of these is the relationship between 
grade level and the magnitude of correlations between pre- and posttest 
scores. This relationship la almost perfect for grades one through six 
(rho a .1.0 for reading in saturated and unsaturated schools and for 
math in saturated schools' rho = .94 for math In unsaturated schools). 
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When pooled across subject matters and types of school*, the pre-posttest 
correlations are as follows: 

r * .259 
r a .390 
r = .584 

r = .6< : • ' . 

r ^ .718 / 1 

r - .841 , 

It should also be rerae^ared that these correlations are based 
on a restricted range of talent (disadvantaged childran) and would pre- 
Bumably be higher if based on a representativa aample of California 
school children. The implicatloni are clear r whatever differences 
existed in project expenditures or treatments had very little impact on 
the participating children -- at least at the higher grade levels / 
Post'- treatment performance is highly predictable from pre-treatment 
status * 

Other notable between=grade differences werei (a) rather high 
correlations (,65 in unsaturated and >52 in saturated schools) between 
math gains and number of project participants per ichool or district for 
the sixth grade only (the corresponding correlations were near zero at 
other grade levels)! (b) mcKlerately high correlations (.35, ,32, ,36) 
between pretest scores and all categories of supplementary expendttures 
Q^iy in sixth grade reading projects in saturated achools whereas these ' 
correlations tended to be near zero elsewhere^ . and. (c) generally greater 
deviations of grades l, and 7 through 12, correlations from the weighted 
averages presented in Tables 6 through 9 than was the case in grades 2 
through 6* 

ThS first two. of these three between-grade differences do not seem 
to be interpretable in the light of the data in hand* The third 



* This is not a strictly legitimate procedure here since some of the 
same pupils will be represented in more than one of the pooled corre^ 
lationsj hence they will not be independent. On the other hand, the 
procedure is useful for Illustrative purpoees, 
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First grade 
Second grade 
Third grade 
Fourth grade 
Fifth grade 
Sixth grade 
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difference is most likely attributable to smaller (and probably less 
representative) sample siEes. 

Defipite these observed differences, correlations were pooled across 
grade livels using Fisher's £ transformation (McNemar, 1969, p, 158), 
The matrices of pooled correlations are presented in Tables 6 through 

. TABLE 6 

Pooled Correlations — Reading Projects in Unsaturated Schools 





Pre- 


Post- 


Aver. Mo . 


No, of 


Reg. 


iitle 


Title I 




test 


test 


Gain 


Pupils 


$ 


I $ 


+ STEP 




2 


3 


4 


5 


6 


7 


8 


2 


1.000 














3 


.646 


■ 1.000 












4 


-.258 


.521 


1.000 










5 


-.025 


.042 


.058 


1.000 - 








6 


.101 


.060 


-.049 


.027 


1.000 






7 


-.053 


.008 


.061 


-.042 


-.073 


1.000 




8 


-.052 


.006 


.057 


-.043 


-.071 


.999 


1.000 


g* 


-.056 


-.023 


.034 


-.057 


-.059 


. 811 


.815 


* 


Variable 


9 is Total 


Suppiementary Per-pupll Ejipendltures 


in Tables 6-9 








TABLE 7 










Pooled 


Correlation 


s -- Reading 


Projects 


in Saturated Schools 




Pre- 


Post- 


Aver, Mo. 


No. of 


Reg. 


Title 


Title 1 




test 


test 


Gain 


Pupils 


$ 


I $ 


+ STEP 




2 


3 


4 


5 


6 


7 


8 


2 


1.000 














3 


.666 


1.000 • 












4 


-.109 


.647 


1.000 










5 


.116 


.146 


.083 


1.000 








6 


.059 


.045 


-.012 


-.027 


1.000 






7 


.079 


.226 


.272 


.030 


.265 


1.000 




8 


.102 


.197 


.258 


.013 


.277 


.988 


1,000 


9 


.096 


.237 


.273 


-.006 


,448 


.856 


.858 



o ' 1," 
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TABLE 8 

Pooled Corralations — Math Projacts in Unsaturated Schools 





Pre- 


Post- 


Aver, Mo. 


No. of 


Reg. 


Title 


Title I 




test 


test 


Gain 


Pupils 


$ 


I $ 


+ STEP 




2 


3 


4 


5 


6 


7 


8 


2 


1.000 












3 


.665 


1.000 












4 


-.185 


.590 


1.000 










5 


,129 


.153 


.072 


1.000 








6 


.019 


-.013 


-.048 


.028 


1.000 






7 


-.023 


.007 


.032 


.027 


-.069 


1.000 




8 


-.027 


,002 


.029 


.024 


-.064 


1.000 


1.000 


9 


-.011 


-.017 


-.008 


.018 


-.034 


.847 


.848 



TABLE 9 

Pooled Correlations Math Projects in Saturated Schools 





Pre- 


Post- 


Aver, Mo. 


No. of 


Reg. 


Title 


Title I 




test 


test 


Gain 


Pupils 


$ 


1 $ 


+ STEP 




2 


3 


4 


5 


6 


7 


8 


2 


l.OuO- 












3 


.601 


1.000 










/ 


4 


-.070 


.660 


1.000 








5 


.203 


.103 


-.027 


1.000 








6 


.030 


.023 


-.026 


-.007 


1,000 






7 


.328 


.212 


-.015 


-.024 


.241 


1,000 




8 


,312 


.200 


-.022 


.042 


.258 


.981 


1.000 


9 


.325 


.216 


-.012 


.025 


.430 


.921 


.931 



In all four matrices, the highest correlations are those interrelating 
the three supplementary expenditure variables a not surprising finding 
since Title I eKpenditures dominate all three. Next in mainitude are the 
correlations of pretest with posttest and of posttest with average monthly 
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gain. With these exceptions, all other correlations were near zero for 
botn reading and math in unsaturated schools. In the saturated schools 
there were correlations of the order of ,2 or .3 be&^een supplementary 
expenditure variables and achievement meaaures. Again, for reasons chat 
are not clear, the achlevement^expenditure correlations involved pre^ 
and postteet scores for math but postteat and gain scores for -reading. 
The latter relationship is compatible wich normal expectations while the 
former implies that funds were spent in in verse proportion to the needs 
of participating pupils, 

Wargo et al, (1972) correctly pointed out that since Title I funds 
. supplement regular per-pupil eKpenditures, their impact may be obscured 
by thB large variaLious known to exist in regular expendituirfcs , Thu 
need to account for this relationship is particularly evident in 
California, where published regular per-pupil expenditures by school 
district ranged from $353,26 to $3,017.08 in FY 1971. Fortunately the. 
technique of partial correlation affords a convenient method for examining 
the impact of. Title I eKpenditurea on achievement with the effects of 
variations in regular eKpenditures statistically removed, 

A second problem which can be dealt with at least to some extent 
by means of partial correlations concerns the use of gain scores as 
criterion variables. This issue has been widely debated in recent years 
(Cronbach Furby, 1970| 0*Connor, 1972), As those concerned with this 
issue point out, change scores are unreliable. Many recommendations for 
alternatives have been made including the use of posttest scores only, 
various types of residual change scores, covariance analysis, etc* 
While the data available to this study do not lend themselves to all 
of these approaches, it was possible to "partial out" the effects of 
pretest score differences from the relationships between gain scbres and 
expenditure variEbles. It would also have been possible to partial pre= 
test score differences out of posttest score relationships but, as 
Shaycoft (1967, Appendix H-2) has shown, these two approaches are mathe- 
matically equivalent and will yield, identical results. 



ERLC 



21 



Tables 10 through 13 present correlations between gains and supple- 
mentary per-pupll expenditures with reiular per-pupil expenditures and 
pretest scores partialled out. These statistics are presented separately 
by grade level within each of the four analytic groups . Sums or weighted 
averages (computed using Fisher's z transformation) are presented at 
the bottom of each table. 

Tables 10 through 13 present, for the first time in this report, 
statistics on the number of schools or districts for which some data 
were available and the number of schools, or districts for which complete 
data were available. As can be seen, the attrition rates were quite 
high in general and about twice as great for the saturated schools (72% 
for both reading and math) as for the msaturated districts (30% for 
reading and 31% for math). 

The preponderance of the losses was due to the Inclusion of only 
pre- or only posttest data which precluded the computation of gain 
scores. Occasional losses occurred because the interval between tests 
was not specified, because no regular per-pupil expenditure data were 
available, and, in two instances, because the data were obviously 
erroneous (Title I per-pupil expenditures of $12,000 and $32,000 
respactlvely) , 

The difference in attrition rates of saturated schools as opposed 
to unsaturated districts Is attrlbutod to the fact that multi-school 
districts would "survive" even when schools within them were lost. 

There do not appear to be any systematic effects due to grade level 
although where samples are small, and probably non-representative (grades 
1, and 7 through 12), deviations from the weighted averages are some- 
times substantial. 

The weighted average partial correlations included in Tables 10 
through 13 are presented again in Table 14 to facilitate betwaen-group ' 
comparisons. Also included in Table 14 are the corresponding multlpl 
correlations . Both the partial and the multiple correlations are 
generally low with the largest one (.313) accounting for less than IQ% 
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TABLE 10 



Correlations* between Gains and Supplenientary Expenditures 
/ with Regular Expenditures and Prettst Scores Partialled 

Out ~ Reading in Unsaturated SchQols 



Grade Level 


* r 
W 




^47-6 


^49-6 


^47.2 


'"47.26 


1 


248 


204 


.066 


-.001 


.033 


.072 


2 


485 


316 


.126 


.064 


.128 


.124 


3 


488 


321 


-.006 


-.008 


.021 


-.018 


4 


473 


312 


-.063 


-.024 


-.070 . 


-.075, 


5 


464 


298 


.114 


.038 


.110 


.106 


6 


443 


277 


.172 


.124 


.170 


.172 


7 


54 


49' 


-.071 


.054 


-.027 


-.098 


8 


51 


46 


-.083 


-.050 


-.033 


-.080 


9 


81 


77 


.127 


.066 


.133 


.143 


10 


62 


61 


-.082 


.010 


-.089 


-.108 


11 


52 


51 


-.019 


.011 


-.043 


-.124 


12. 


37 


36 


.039 


-.001 


.021 


-.056 


Totals 


2938 


2048 


.060 


.031 


.057 


.046 


*N = Nunier 


of districts which 


subinltted 


reports 






N_ = Number 
K 


of districts with 


complete 


achieve inant 


and cost 


data 



^47*6 ^ Correlation between average monthly gains and Title 1 per-pupll 
expenditures with regular per-pupil expenditure- .pnrtialled out 

^49-6 ^ Correlation between average monthly gains and l:^vn] supplementary 
per-pupil expenditures with regular par-pupil axnnndlturas par- 
tialled out 

^47^2 ^ Correlation between average monthly gains and Title I per-pupil 
expenditures with pretest scores partialled out ■ 

^47' 26 " Correlation between average monthly gains and Title I per-pupil 

expenditures with pretest scores and regular per^pupil expenditures 
partialled out • 
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TABLE 11 

Correlations* between Gains and Suppleman tary Expendi Cures 
with Regular Expenditures and Pretest Scores Partialled 
Out Reading in Saturated Schools 



Grade Level 


N • 


N- 
R 


r - 
47^6 


49-6 


-47.2 


47-26 


1 


268 


44 


,117 


-.003 


,111 


.186 


2 


518 


169 


.511 


.489 


,519 


. .672 


3 


552 


167 


All 


.470 


,522 


.607 


4 


505 


161 


.061 


.15 7 


.064 


.046 


5 


495 


166 


,284 


.352 


.287 


.308 


6 


473 


166 


,046 


. 141 




. wu ^ 


7 


65 


5 


, 305 


-.844 


.990 


.491 


8 


59 


5 


.528 


-.201^ 


.646 


.741 


9 


96 


4 


.163 


.391 


.023 


.000 


10 


77 


1 










11 


63 


1 










12 


45 


1 










Totals 


3216 


890 


.283 


. 312 


.289 


. .278 


*N = Number 

N„ = Number 

R 


of schools which 
of schools with 


submit tad 
complete 


reports 
achievement 


and cost 


data 



■47.6 



"49.6 



"47.2 



"47.26 
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Correlation between average monthly gains and Title I per^-pupil 
expenditures with regular per-pupil expenditures partialled out 

Correlation between average monthly gains and total supplementary 
per-pupll expenditures with regular per-pupll eKpenditures par^ 
tialled out . 

Correlation between average monthly gains and Title I per-pupil 
expenditures with preteot scores partialled out 

- Correlation betT^een average monthly gains and Title I per-pupil 
expenditures with pretest scores and regular per-pupil expenditure 
partialled out 



24 



TABLE 12 



Correlations^ between Gains and Supplementary Expenditures 
with Regular EKpenditures and Pretest Scores Partialled 
Out " Math in Unsaturated Schools 





a 


H 


^47.6 


^49.6 


^47-2 


'^47.26 


1 


246 


194 


-.008 


-.110 


.034 


0.038 


2 


373 


281 


.019 


-.013 


.005 


-.004 


3 


463 


292 


.009 


-.073 


,012 


-.011 


4 


459 


289 


.007 


-.022 


-.001 


-.007 


5 


445 


278 


.040 


.024 


.042 


.035 


0 


/i "3 ^ 


267 


,085 


.070 


,111 


.106 


7 


50 


46 


-.025 


-.078 


-.064 


-.068 


8 


44 


40 


-.212 


-.237 


-.205 


-.210 


9 


69 


65 


.072 


.091 


.102 


.073 


10 


47 


46 


.218 


.265 


.186 


.191 


11 


33 


32 


.020 


-.033 


.022 


-.005 


12 


20 


13 


.276 


.225 


.221 


.257 


Totals 


2684 


1849 


.029 


-.011 


.033 


.030 


^ Number 
- Number 


of districts which 
of districta with 


submitted 

complete 


reports 
achievetnent 


and cost 


data 



^47.6 ^ Correlation betwean average monthly gains rad Title I per^pupil 
expenditures with regular per-pupil aKpenditures partialled out 

^49 ^ Correlation between average monthly gains and total supplementary 
per^pupil eKpendltures with regular per^pupil eKpenditures par- 
tialled out 

^47^2 ^ Correlation between average monthly gains and Title 1 per^pupil 
expenditures with pretest scores partialled out 

^47*26 ^ Correlation between averaga TOnthly gains and Title I per-pupil 

expenditures with pretest scores and regular per--pupll expenditures 
partialled out 
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TABLE 13 
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Correlations'^ between Gains and Supplementary Expenditures 
with Regular EKpenditures and Pretest Scores Partialled 
Out — Math in Saturated Schools 



Grade Level 




\ 


^47-5 


^49.6 


^47 = 2 


^47.25 


1 


267 




wHq / 




.652 


,663 


2 


HKJD 




^* 110 


--.101 


-.055 


-.063 


3 


496 






— *uo6 


-.017 


-,015 


4 


491 


iVO 


.023 


--,003 


-.049 


,004 


5 


476 


167 


-.018 


,024 


-.040 


-.029 


6 


J J 


J. J o 


— .uoy 


-.047 


-.099 


-.10 7 


7 


59 


4 


.481 


.172 


.613 


.999 


8 


53 


4 


.602 


.311 


.907 


.999 


9 


83 


4 


.967 


.967 


-.990 


-.999 


10 


56 


1 










11 


40 


1 










12 


25 


1 










Totals 


2886 


805 


-.001 


,003 


.002 


.004 



^ Number of schools which submitted reports 

^ Number of schools with complata achievement and cost data 

^47. 6 ^ Correlation between average monthly gains and Title 1 per-pupil 
expenditures with regular par-pupil eKpenditures partialled out 

-49.6 " Correlation between average monthly gains and total supplementary 
per=pupil expenditures with regular par-pupil expenditures par- 
tialled out 

^47.2 ^ Correlation between average monthly gains and Title I per^pupil 
expenditures with pretest scores partialled out 

^47-26 ^ Correlation between average monthly gains and Title I per-pupll 

expenditures with pretest scores and regular per-pupil eKpendltures 
partialled out 
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TABLE 14 



Partial and Multiple Correlatioiis'* Involying 
Achievement and Expenditure Variables 



• 




f49^6 


^■47.2 


^4 7 -26 




4* 76 


4* 96 




'^4.726 


Reading - Unsat, 


.060 

T075 


.031 
:Q58 


.057 
.262 


.035 
.263 


Reading - Sat. 


.283 


*312 


.289 


.278 


.285 


7312 


.302 


.313 


I-lath - Unsat. 


.029 
,056 


-.011 
,049 


.003 
.187 


.030 
.192 


Math - Sat, 


-,001 
V029 


,003 
-*0i3 


.002 
.071 


.004 
.075 


See Table 13 for 


key to partial 


correlatione 


Corresponding multiple! 


have average monl 


;hly gains as th 


e criterion 


with other 


variables as 



predictors , 



of the criterion variance. All of the correlations are undoubtedly 
depressed by the inadequaciee of the data which were discussed aurlier. 
How high they might rise if better data were obtained cannot ^ unfortun- 
ately, even be estimated from information currently available* 

If there Is a positive relationship between expenditures and gains ^ 
it is apparent from the data in Table 14 only for reading projects in 
saturated achools. While four other multiple correlations lie in the 
range from *19 to *26j they are primarily due to relationships between 
pretest scores and galnsj not eKpendltures and gains. 

As mentioned in the Introduction of this report 5 the so-called 
critical mass issue was to be investigated as one aspect of the cost-- 
effectiveness study. Toward this end the computer was instructed to 
print out scatter diagrams showing the relationships between average 



(d) total per^pupii expenditures. These scatter diagrams were obtained 
at each grade level within each of the four analytical groups. 

The 192 resulting scatter dlagranis were individually eKamlned in 
an attempt to identify some point on the expenditure continua where 
greater than average gains significantly outnumbered below average 
gains. It was clear from these examinations that no meaningful cut- 
off points existed for any of the expenditure variables and no further 
analyses were undertaken. While this decision resulted in an unsatis-- 
factory amount of. negative evidence regarding the critical mass issuer 
the only alternative was to conduct a compleK series of Chi Square (or 
equivalent) analyses, collapsing each of the 192 scatter diagrams in a 
succession of different ways. The amount of time and effort required 
for this task could not be justified In the light of other priorities. 

A decision was also made not to Include all 192 scatter diagrams 
in this report. On the following pages, however ^ five examples are 
presented. The first. Figure 3, is for first grade math in saturated 
schools and shows a positive relationship between average monthly gains 
and Title I per^pupll expenditure (r ^ •478), The relationship appears 
to be more or less constant over the entire range of both variables. 
Figure 4 shows another positive (r ^ .458) relationship beft^een the 
same two variables— this time for third grade reading in saturated 
schools. In this case, however, the correlation seems more directly 
due to the poor performance of children at schools reporting lero 
expenditures. 

The scatter shown in Figure 5 is more typical than those in Figures 
3 and 4 which were specifically selected because they depicted two of 
the highest correlations obtained. The Figure 5 diagram shows a near 
zero (r ^ -,026) relationship between average monthly gains and total 
per-pupil expenditures for third grade reading projects in unsaturated 
schools. It was selected because it represents the largest number of 
schools at any wl thin-group grade level (N - 321) , 
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DISnUSSION AND CONCLUSIONS 



The analyses dascrlhed in the preceding chapter of this report do 
not provida much support for the exxstence of a relationship betoeen 
achlevamant gains and per-pupil eKpendltures elthar regular or 
supplemental. For reading programs In saturated schools there did 
appear to ba a cost=benefit relationship involving Title 1 eKpendltures 
which accounted for approximately 10% of the achievement variance. This 
relationship, as was pointed out earlier, might be significantly stronger 
if better measures were obtained of the dependent and independent var- 
iables. On the other hand, no cos t--benef it relationships were identi- 
fied involving mathematics programs or reading programs In unsaturated 
" schools. 

No evidence was found supporting the existence of a ''critical 

mass. 

Children In unsaturated schools were found to TOke greater gains 
In both reading and math than did their counterparts in saturated 
schools. Since supplementary (particularly Title I) expenditures were 
greater In unsaturated schools ^ it appeared, at first, that a relation- . 
ship might be. found between expenditures and gains. Within the unsat= 
urated schools, however^ no such relationship could be faund even 
though there was a wide range of expenditures which substantially over- 
lapped that of saturated schools. Furthermore j there was some evidence 
suggesting that the greater gains made by children in unsaturated schools 
during the school year were offset by greater losses during the summer. 

With respect to this latter point, none of the data enalyzed covered 
the same children for more than a single school year. Inferences made 
regarding summer losses were based on the fact that pretest grade- 
equivalent scores were consistently at or below the disadvantaged norm 
(.7 X grade level) while gains each school year exceeded month for 
month. At no grade level were Title I participants in unsaturated 
schoola significantly batter off than their'peers in saturated schools. 

Clearly this evidence and the Inferences drawn from it are Important 
for^ both educational policy and educational practice. The evidence is 



only circuma tan tialp however^ and needs to be verified by means of lon- 
gitudinal data collected over at least a IZ^month period. Collection 
of such data would appear to.be a high priority activity in view of the 
great Importance of policy decisions in this area* 

The California Follow=up study found quite a high correlation be- 
tween costs and benefits In the sample of Title I projects It eKamlned, 
It should be pointed out that comparable results would have been obtained 
here had data from saturated and imsaturated schools been comblneu as 
presumably they were in that study. E<ren if the within^group correlations 
had been ^ero for both saturated and unsaturated schools, a fairly strong 
(but spurious ) relationihip would reiult from pooling the two groups 
since one was substantially higher than the other in terms of both cost 
and achievement variables. Figure 8 illustrates this phenomenon. 



Per-pupll 
Expenditures 




Achievement Gains 



Fig. 8, Production of a Spurious Correlation by the Pooling of Two Non- 
homogeneous Groups within Each of which the Correlation is Zero, 

The fact that unsaturated school districts report both higher gains 
and higher expendlturei than iaturated gchools suggeste, in the absence 
of a significant cost-benefit correlatlpn, that the difference bet 



types of schools is produced by some other influence. Socioeconomic 
status md racial mixture represent two plausible explanatory hypotheses , 
but the data in hand do not lend themselves to Investigation of this issue 

In addition to the zero-order correlations calculated between gain 
scores and the various supplementary expenditure categories, partial 
correlations were calculated to remove the effects of variations In reg-- 
ular par-pupil expenditures and pretest grade^equivalent status. This 
procedure raised the galn^Title I expenditure correlations very slightly 
for reading projects in saturated schools but had no noticeable effect .. 
In the other groups analyzed. Apparently the impact of Title I expendi- 
tures is largely independent of regular per=pupil expenditures. This 
finding was somewhat surprising in view of the seemingly sound expecta- 
tion expressed by Wargo et al. (1972) that the effects of Title I expen- 
ditures would be at least partially determined by the level of regular 
per^pupil expenditures . 

Multiple correlations were also calculated using achievement gains 
as the criterion with pretest scores and various experidlture categories 
as predictors* These multiple correlations ware not much different from 
the corresponding zero-order correlations between gains and Title I ex- 
penditures except for reading and math projects in unsaturated schools 
where substantial increases were observed. These Increases were attribu- 
table to a combination of substantial negative correlations between pre- 
test and gain scores and slightly negative correlations between pretest 
scores and Title 1 expenditures , While this finding was not of particu- 
lar significance J it directed attention to the rather high positive corre- 
lation (r ^ ,328) between pretest scores md Title I expenditures for 
math projects in saturated schools. This correlation signified a district 
tendency to spend the least money on the children with the greatest edu- 
cational need Bnd vice versa. 

As was discussed elsewhere in this report, the critical mass issue 
was addressed by visual examination of 192 scatter diagrams showing each 
school's (or district's) position on a two-dimensional plot defined by 
average monthly gain and per-pupil eKpenditure (regular , Title I, Title 
1 plus STEP, and total per-pupil expenditures were plotted separately 



against gain scores). No evidance could be found suggestini that thera 
was a specific point on any of the expanditure continua above which graatar 
Chan average gains outnu^ered below average gains. Positiva relationships 
between gains and expendituras were identified imder some conditions (sea 
above) but there never appearad to be the kind of discontinuity implied 
by the critical mass concept. 

Many states ^ including California^ have accepted the critical mass' 
concapt and have raandatad specific minimum per-p up il expenditures. Funds 
appropriated for Title I programai however ^ have been grossly inadequate 
to serve all eligible children at levels equal to or above the critical 
' mass criterion. As a result, Miny eligible children have not been served 
(California served approximately 401 o£ the eligible children during 
FY 72). Failure to serve the other eligible childrtn could become a 
significant moral and political issue should the critical mass concept 
fail to stand up under scrutiny. 

In concluaion it may be said that this study produced some evidence 
supporting the existence of a positive relationship between supplementary 
per-pupll expenditures and achievement gains. This relationship was 
obsarved under only one of four treatraent conditions 5 but the character- 
istics of the data may have obscured siTnilar relationships under the other 
treatment conditions just as it must be assumed they attenuated the size 
of the obierved correlation. 

On the other handj correlations between pre= and posttest scores 
were quite high under all treatment conditions and increased in magnitude 
as a function of grade level. Poit--treatment median grade-equlvalent 
scores maintained much the same relative standing from echool to school 
and district to district as the corresponding pre-treatment scores ~ 
a finding which implies that all projects were approximately equally affec- 
tive in prbducing achievement benefits. It would also tend to suggest 
that treatment impact had not been great , as even small differences among 
highly effective programs would tend to be too disruptive of the status 
quo for the. correlations between pre- and posttest sc^ to be as high 
as were observed, . 
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